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ABSTRACT 

The suggestions offered in this paper are based on 
the experiences of two scientific researchers of how evaluations 
undertaken in a competitive arena, in which true experimental designs 
are not viable, can be designed in such a way that meaningful 
comparative data can be examined. Case studies of the Science and 
Engineering Research Semester and the Laboratory Graduate Research 
Participation programs at the Oak Ridge Institute for Science and 
Education illustrate how one can make use of existing comparison 
groups when the establishment of control groups for each cohort in an 
educational study is not economically feasible. Data from national 
surveys are used for comparative purposes. These surveys include: (1) 
the Survey of Earned Doctorates of the National Science Foundation 
(NSF); (2) the Survey of Doctorate Recipients, another NSF survey; 
(3) the NSF New Entrants Survey; (4) the Survey of Experienced 
Scientists and Engineers, also an NSF component survey; and (5) the 
National Survey of College Graduates. Items from these surveys that 
are clearly relevant are selected for comparisons. In conclusion it 
is remarked that this approach is not a panacea, and requires 
thorough understanding and careful evaluation of the national survey, 
but it can be a useful and cost-effective alternative to traditional 
control-group designs. Ten figures illustrate survey use. (Contains 
nine references.) (SLD) 
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Using National Surveys to improve the Efficiency and Effectiveness 
of Broad-Based Program Evaluations 1 

Under ideal circumstances-with no budgetary, time, procedural, or personnel 
constraints-most researchers would choose to employ a true experimental 
design when conducting program evaluations. Such designs, if properly 
conceptualized and carefully implemented, allow the researcher to reach 
defensible conclusions about the existence and extent of programmatic effects. 
Unfortunately, many situations exist in which the use of a true experimental 
design is either impossible or impractical. These situations include such factors 
as ethical constraints (e.g., problems of withholding positive treatments), 
sampling constraints (e.g., inability to randomly select and/or randomly assign 
subjects to treatment groups), and the potential contamination of the treatment 
and control groups. 

In the context of assessing the outcomes of national educational programs in a 
competitive environment (i.e., in which grants or contracts are awarded based on 
solicited proposals), the considerations are apt to be much more mundane. If a 
researcher proposed to implement an aggressive, empirically sound, true 
experimental evaluation design, his/her proposal would probably not be 
financially competitive with proposals that use less rigorous evaluation designs. 
The likelihood of its inherent scientific superiority being valued highly enough by 
the sponsoring agency to warrant its being funded instead of a less expensive, 
less rigorous alternative, is not at all predictable. Often sponsors are 
(understandably) more concerned with showing the positive outcomes of a 
program (so that they can justify its continuation) than they are with determining 
which of these outcomes may be clearly attributable to the program. Because 
positive outcomes are often easy to identify and frequently seem to be 



1 Thi$ paper is based on a presentation made at Evaluation '91, the annual meeting of the American Evaluation Association. 
Work on the present paper was supported, in part, by the American Statistical Association and the National Science 
Foundation. The paper is based on activities supported by the U.S. Department of Enerqy. Office of Science Education and 
Technical Information, and performed by the Oak Ridge Institute for Science and Education (ORISE). Oak Ridge Associated 
Universities manages and operates ORISE under the U.S. Department of Energy Contract No. DE AC05-760R00033. Any 
opinions, findings and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily 
reflect the policies and views of the American Statistical Association, th; National Science Foundation, the U.S. Department of 
Energy, the Oak Ridge Institute for Science and Education. Oak Ridge Associated Universities, or the University of Tennessee. 
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acceptable indicators of the worth of a program in the eyes of those who fund 
them, the program sponsors are often reluctant to commit funding to 
sophisticated evaluation procedures that could otherwise be spent on the 
program itself. 

The extent to which a professional evaluator is willing to develop and undertake 
a program assessment that is less scientifically rigorous than he/she might like is 
a highly personal matter, but this issue will not be considered here. The realities 
of contract evaluation often dictate that compromises be made in "best practice" 
techniques. This does not mean that competitively-won evaluations are 
necessarily inferior evaluations. Nor does it mean that program effects cannot 
be established for evaluations that do not use true experimental designs. The 
purpose of this paper is to offer suggestions, based on the experiences of the 
authors, on how evaluations undertaken in a competitive arena in which true 
experimental designs are not viable can be designed in such a way that 
meaningful comparative data can be examined. This approach is consistent with 
Pattern's (1982) notion of practical evaluation, and is responsive to his call for 
generating "a great deal of really useful information with extremely scarce 
resources" (p. 19). 

Making Use of Existing Comparison Groups: A Case Study 

Context 

The Oak Ridge Institute for Science and Education (ORISE) is a U.S. 
Department of Energy facility, which is managed and operated by Oak Ridge 
Associated Universities, a private, not-for-profit corporation sponsored by 65 
colleges and universities. Through its Science/Engineering Education Division, 
ORISE conducts educational program evaluation and assessment studies for a 
variety of federal sponsors, including the Department of Energy, the Department 
of Education, and the National Science Foundation. In general, the objectives of 
the evaluation and assessment activities include providing quantitative and 
qualitative measures of the impacts of these programs on participants, 
assessing programmatic achievements, providing information for the 
improvement of program operations, and determining the extent to which the 
programs meet their stated objectives. The educational programs are intended 
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to enhance some element of the production of scientists and engineers (S/Es) in 
this country-for example, the recruitment and retention of students to S/E 
programs of study and S/E careers; the pursuit of graduate S/E degrees; the 
extent of research involvement by scientists and engineers; or addressing the 
current underrepresentation of women and minorities in S/E study and S/E 
careers. 

One such program is the Science and Engineering Research Semester (SERS) 
program, which is sponsored by the Office of Science Education and Technical 
Information, U.S. Department of Energy (DOE). The SERS program provides 
research appointments to about 150 U.S. undergraduate students each year at 
one of six DOE national laboratories. Participants have the opportunity to 
become involved in "hands-on" research, working with scientific teams engaged 
in long-range investigations and using state-of-the-art facilities and equipment. 
SERS research appointments are available in biomedicine, chemistry, materials 
science, engineering, physics, environmental science, geoscience, mathematics, 
computer science, artificial intelligence, energy systems, and waste technology. 
One of the objectives of the SERS program is "to encourage students to seek 
graduate degrees and research careers in science and engineering disciplines 
or areas supportive of the DOE mission" (Stevenson et al., 1991 , p. 8). 

Another such program is the Laboratory Graduate Research Participation (Lab 
Grad) program. The Lab Grad program is an important component of the 
University/DOE Laboratory Cooperative Program, which is also supported by the 
Office of Science Education and Technical Information in DOE. The program 
enables about 200 graduate students annually io conduct thesis or dissertation 
research in residence at a DOE facility. Lab Grad participants work with 
laboratory scientific staff and are able to make use of equipment and facilities 
that are not generally available on university campuses. Participants' research 
projects must comp'ement ongoing research at the host facilities and must meet 
degree requirements for the students' graduate programs. Among the stated 
objectives for the Lab Grad program are "to encourage graduate students to 
pursue careers and to continue to work in areas supportive of the DOE mission" 
(Vivio & Stevenson, 1990, p. 4). 
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Need for Comparison Groups 



The SERS and Lab Grad programs are typical of educational programs for which 
the authors have been conducting follow-up essments for several years. The 
problems inherent in evaluating the extent » o which these programs meet 
general (but clearly primary) objectives such as those above are considerable. 
The measurement of short-term effects (e.g., pre- and post- research-experience 
attitudinal comparisons) are of very limited use, as far as determining whether 
participants will indeed go on to pursue graduate degrees or research careers in 
science or engineering. Long-term follow-up of former participants is essential 
to reaching firm conclusions about the attainment of such objectives. The 
special difficulties in maintaining current addresses for students and early-career 
graduates are well established. Assuming that some former participants do 
indeed pursue graduate degrees and/or research careers in science or 
engineering, it is difficult to attribute those choices to the SERS or Lab Grad 
programs without some basis for comparison. Identifying, surveying, and 
following-up a control group of non-SERS or non-Lab-Grad students for each 
cohort of SERS or Lab Grad participants is economically infeasible for these 
programs. 

Identification of Appropriate Comparison Groups 

In order to assess the degree to which programs like SERS or Lab Grad have 
met these general objectives in the absence of traditional control groups, 
researchers at ORISE and Argonne National Laboratory have turned to several 
national data bases containing information on scientists and engineers. In 
cooperation with the National Science Foundation (NSF) and the National 
Research Council, questionnaires have been developed in which key items used 
in the evaluation of the programs being evaluated conform to those used in large 
national studies sponsored by NSF, DOE, and other federal agencies. Using 
questions and response options from these national surveys makes it possible to 
compare program participants to national norms with respect to many relevant 
variables. 

Depending on the objective being assessed and the point in time at which the 
assessment is made (relative to the completion of participation in the 
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educational program), items from one or more of the following national surveys 
(for which NSF is the primary sponsor) are used: (a) the Survey of Earned 
Doctorates, (b) the Survey of Doctorate Recipients, (c) the Sbivey of Recent 
College Graduates (formerly called the Survey of Recent Science, Social 
Science, and Engineering Graduates-or the New Entrants Survey), and (d) the 
National Survey of College Graduates (formerly called Vr*i National Survey of 
Natural and Social Scientists and Engineers-or the Survey of Experienced 
Scientists and Engineers, or the Postcensal Survey). Each of these surveys is 
described below, based on the data collection procedures used in the 1980s and 
those presently in use. (The Divii ion of Science Resources Studies at NSF 
undertook a major restructuring of these data collection efforts for the 1990s, 
which substantially changed the characteristics of several of the surveys.) 

Survey of Earned Doctorates . This survey is conducted annually for NSF by the 
National Research Council to collect information on the number and 
characteristics of recipients of doctoral degrees in the United States. The data 
gathered in this survey are used in the construction of the Doctorate Records 
File (DRF), which is virtually a census of all recipients of research doctorates 
(excluding professional or clinical degrees such as the J.D., M.D., and D.V.M.) 
awarded by U.S. educational institutions since 1920. The DRF contains more 
than 1,000,000 records, almost 85 percent of which came from the Survey of 
Earned Doctorates, which has been conducted since 1958. (National Science 
Foundation [NSF], 1987, pp. 15-1 9; Ries & Thurgood, 1993) 

Survey of Doctorate Recipients . The Survey of Doctorate Recipients is one of 
three NSF surveys that cover various S/E subpopulations. Together, these three 
surveys comprise NSF's Scientific and Technical Personnel Data System 
(STPDS). Conducted biennially for NSF by the National Research Council since 
1973, the Survey of Doctorate Recipients is designed to provide national 
estimates of the supply and utilization of science and engineering doctorates. 
This longitudinal survey is based on a sample drawn from the DRF, and is 
stratified on several characteristics (such as sex, field of doctorate, and a 
combination of racial/ethnic identification, handicap status, and nativity). Data 
are collected for major demographic and employment-related variables. 
Demographic variables include age, citizenship, marital status, sex, race, and 
ethnicity. Employment-related variables include employment status, employment 



r 



sector, primary work activity, and salary. (Holmstrom, 1988; NSF, 1987, pp. 21- 
25; Office of Scientific and Engineering Personnel, National Research Council 
[OSEP/NRC], 1993, pp. 3-9) 

The SDR sample size in 1989 was approximately 74,000, which represented 
about a 1-in-13 sample of S/E doctorates, but the response rate during the five 
survey cycles in the 1980s deteriorated to 55 percent in 1989. In 1991 , the 
sampling frame was modified somewhat, and the sample size was reduced to 
about 38,000, so that survey resources could be re-directed to improving the 
response rate. The improvement in response rate was considerable in 1991 -to 
80 percent (87 percent, when weighted), which includes data gathered in a CATI 
effort. (OSEP/NRC, 1993, pp. 3-9) 

New Entrants Survey . A component of the STDPS, the objective of this biennial 
survey is to provide data on the demographic and employment characteristics of 
individuals who receive bachelor's or master's degrees in S/E fields from U.S. 
institutions. In the 1980s, this was a cross-sectional mail survey, conducted for 
NSF by the Institute for Survey Research at Temple University. In the 1990s, 
the cross-sectional portion of the survey, which is now conducted primarily as a 
CATI survey, will be carried out by Westat, Inc., and a sample of each 
graduating class will be added to the National Survey of College Graduates, 
which is described below. 

The survey population is limited to S/E degree recipients who were citizens or 
permanent resident alien: at the time of their degree award. A two-stage 
probability sample is used, with the primary sampling unit being universities and 
colleges, stratified by geographic region, public/private institutional status, type 
of curriculum offered, proportion of graduates with S/E majors, and two special 
strata consisting of universities and colleges that have a predominantly black 
student body or that have high concentrations of Hispanics. The secondary 
sampling unit consists of graduates drawn from the sample of universities and 
colleges. Individuals drawn from institutions in the special strata are 
oversamplec! in order to increase the reliability of data on racial/ethnic groups. 
Like the Survey of Doctorate Recipients, data are collected for major 
demographic and employment-related variables. Demographic variables include 
age, citizenship, marital status, sex, race, and ethnicity. Early career 
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employment-related variables include employment status, employment sector, 
primary work activity, and salary. (Holmstrom, 1988; NSF, 1987, pp. 27-39) The 
1993 survey has been expanded to include more extensive information on 
educational background, educational choices, professional development 
activities, and parents' education. 

The sample size for the 1990 survey, which included the graduating classes of 
1988 and 1989, was about 26,000. The 1993 survey, which includes the classes 
of 1990, 1991, and 1992, has a sample size of approximately 28,000. This 
represents about a seven percent sample of master's-level S/E degree awards 
and a four percent sample of bachelor's-level S/E degree awards. (J. Tsapogas, 
personal communication, September 22, 1993) 

Survey of Experienced Scientists and Engineers . An integral component of the 
STPDS, this biennial longitudinal survey provides data on the number and 
characteristics of individuals who were identified as being part of the S/E 
population following the preceding decennial census. The survey is carried out 
for NSF by the U.S. Bureau of the Census. The original sample of about 
138,000 used in the 1980s (i.e., based on the 1980 decennial census) was 
stratified on the basis of education, occupation, sex, and race. Information was 
collected on education and training (level and field of degree), demographic 
characteristics (age, citizenship, marital status, sex, race, handicapped status, 
and ethnicity), employment status, and employment profile (occupation, type of 
employer, primary work activity, salary, work experience, etc.). (NSF, 1987, pp. 
3-13) 

The content of the National Survey of College Graduates, first used in 1993, has 
been expanded to include more detailed information about respondents' present 
and former jobs, reasons for changing jobs, training and professional activities, 
educational background, and parents' education level. The 1990s surveys are 
based on a sample of about 214,000 from the 1990 decennial census. Eighty 
percent of these individuals dia not report a S/E occupation on the census form, 
but the initial postcensal survey (which is currently under way) is designed to 
determine which of these individuals hold S/E degrees. Those that do hold S/E 
degrees will be included in subsequent survey cycles, along with the roughly 
43,000 individuals in the sample who reported an S/E occupation in 1990. This 



experienced sample will be supplemented each survey cycle with a sample of 
bachelor's and master's graduates who received S/E degrees since 1990. This 
sample will be drawn from the New Entrants Survey. (M. Regets, personal 
communication, August 1993) 

Selection of Items 

Only those items from the "parent" surveys that are clearly relevant to the 
objectives of the program being assessed are selected for inclusion in the 
instruments for the programs being assessed. These may include general 
demographic items (e.g., gender, race, citizenship) as well as items that relate to 
specific program-related characteristics (e.g., academic major, degree level, 
employment specialty). Demographic differences between the two survey 
populations (i.e., program participants and NSF survey population) may result in 
rival hypotheses about program effects, so it is useful to have the same 
response options for these demographic variables in both instruments if at all 
possible. 

The comparisons to be made from the use of such existing data clearly cannot 
generate conclusions as definitive as those drawn from a true experimental 
design. In order to increase the relevance of these comparisons, nevertheless, 
it is important to minimize the number and severity of uncontrolled influences on 
the comparisons. The careful selection and construction of items on the new 
instrument is critical to maintaining the credibility and usefulness of the 
comparisons. 

Figures 1 and 2 are examples of seemingly straightforward demographic items 
that vary from one survey to another. Because respondents are influenced by 
the choices they are offered, it is important that the researcher pay attention to 
the response options used in the data collection instrument from which the 
comparison group information came. In the case of items for which the 
characteristics being measured may be related to the expected outcomes of the 
evaluation (e.g., when rival hypotheses emerge), the evaluator should ado^t the 
same item wording and response options whenever possible. This means that 
he/she will have to have already identified the comparison group(s) and the 
desired comparisons when the evaluation instruments are constructed. 
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Figures 3, 4, and 5 are examples of items from the NSF-sponsored surveys that 
have been integrated into follow-up assessments of DOE-sponsored programs 
like SERS and Lab Grad. Although the type-of-employer item (Figure 3) is 
rather general in nature, it was important to offer the same response options as 
the national surveys in order to make legitimate comparisons between the DOE 
participants and broader groups of scientists and engineers. The primary-work- 
activity item (Figure 4) and the area-of-national-interest item are more specific 
and are intimately tied to the objectives of DOE-sponsored educational 
programs. 

Designing and Interpreting Comparisons 

Once the data are collected from former program participants using items drawn 
from the NSF questionnaire(s), the precision of the comparisons desired (or the 
precision that is possible) determines how the comparison data are derived and 
presented. In some cases, only very general "benchmarks" are needed in order 
to permit a reader to achieve a reasonable perspective on reported 
characteristics of program participants. It may not matter that the groups are not 
strictly comparable. Published data often suffice for this purpose. When more 
precise comparisons are desired (and both data sets and survey methodologies 
suggest its appropriateness), special tabulations are requested from the NSF 
survey contractor or special tabulations are generated by ORISE from a DOE- 
owned sponsors' data tape. 

Benchmark comparisons . Figures 6, 7, and 8 are examples of comparisons 
drawn from published data and/or special analyses of national data sets that are 
directly related to the objectives of DOE educational programs. In Figure 6, the 
primary work activity of Lab Grad respondents is compared to various groups of 
scientists and engineers. All but one of these comparisons are drawn from 
published data. One has only to consider the difficulty of interpreting-even in 
the most general terms-the data from former Lab Grad participants in the 
absence of the other columns in order to recognize the value of these 
nonequivalent comparisons. Even though each of the comparison groups are 
not comparable to the former Lab Grads in at least some respect (e.g., age, 
degree field, degree level, year surveyed), the distribution of work activities of 
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these groups are nevertheless instructive in gauging the work activities of the 
Lab Grads. The comparisons in Figures 7 and 8 likewise provide a valuable 
backdrop against which the Lab Grads' involvement in federally funded work can 
be viewed. 

More definitive comparisons . The follow-up assessment of former participants in 
the SERS program provided opportunities for deriving more precise comparisons 
by using the data tape of the 1988 New Entrants Survey. The 1988 survey 
includes graduates of the classes of 1985-1986 and 1986-1987. For the 
purposes of the SERS report, a special tabulation was performed using only 
those students earning bachelor's degrees in science or engineering fields 
during the academic year 1986-1987. 

Data for the 1988 New Entrants Survey was collected in the spring of 1988, so 
the 1986-1987 graduates would have been out of school for approximately one 
year. Likewise most individuals who were SERS participants as juniors in the 
academic year 1987-1988 and those who were SERS participants as seniors in 
the academic year 1988-1989 would have been out of school for approximately 
one year in the fall of 1990 when data were collected for the follow-up 
assessment. As a result, this comparison, although imperfect, allows one to see 
how former participants in the SERS program differ from science and 
engineering students who received their bachelor's degrees at about the same 
time. 

Figure 9 depicts a comparison of the academic status of former SERS 
participants and 1987 S/E bachelor's degree recipients. Even though some of 
the SERS participants had not received their undergraduate degrees (meaning 
that the proportion who ultimately attend graduate school is an underestimate in 
the figure), the extent to which former SERS participants were pursuing graduate 

degrees was clearly remarkable compared to the 1987 S/E bachelor's recipients. 

This tendency is directly relevant to the objectives of the SERS program. 

Likewise, the undergraduate degree fields of the SERS participants are more 

consistent with those subsumed under DOE's mission than are those of S/E 

bachelor's recipients in general (see Figure 10). 
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Conclusions 



The techniques described in this paper clearly do not represent a panacea for 
evaluation ills. The use of existing data sources for comparison groups is 
neither helpful nor advisable in many situations. Lifting items from existing 
instruments is not a substitute for thoughtful instrument development. In cases 
in which it makes sense to compare program participants to well-defined external 
populations or to general populations, however, researchers might do well to 
consider using these techniques in planning their evaluations. To maximize the 
interpretability of the results, the comparison group-together with the 
questionnaire(s) that produced the data associated with that group-must be 
identified and evaluated before assessment instruments are designed. This 
requires that the researcher not only acquire the instrument but that he/she have 
a thorough understanding of the methodology (e.g., sampling frame, sample 
design, survey and analysis techniques, response rate, nonresponse 
adjustments) of the data collection effort. Once the researcher understands the 
characteristics of the potential comparison group, he/she must consider the 
limitations of the design nuances and the content of the items themselves with 
regard to the validity, robustness, and relevance of any comparisons that might 
be made. Properly handled, however, the use of existing data for comparison 
purposes can be a useful and cost-effective alternative to traditional control- 
group evaluation designs. 
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Figure 1 . Examples of Variation in Response Options for Demographic 
Items: Race/Ethnicity 



9. What is your racial background? (Check only one.) 




0 O American Indian or Alaskan Native 




1 O Asian or Pacific Islander 




2 □ Black 




3 □ White 




10. Are you Hispanic? □ No □ Yes — > 0 


O Mexican American 


1 


O Puerto Rican 


2 


O Other Hispanic 



from 1990 Survey of Natural and Social Science and Engineering Graduates 

29. Are you: 

1 . □ American Indian or Alaskan Native 

2. O Asian or Pacific Islander 

3. □ Black 

4. □ White 

5. □ Other, PLEASE SPECIFY 



from 1990 Information Sheet for Participants in the U.S. Department of 
Energy Science and Engineering Research Semester Program 

3. What is your racial background? 

1 . □ American Indian or Alaskan Native 

2. d Asian 

3. d Pacific Islander 

4. □ Black 

5. □ Caucasian (Not Hispanic) 

6. O Hispanic 

7. CJ Other 
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Figure 1 (continued). Examples of Variation in Response Options for 

Demographic Items: Race/Ethnicity 



from National Education Longitudinal Study of 1988 (Eighth Grade 
Questionnaire, NELS:88) 

31 A. Which best describes you? (MARK ONE) 

Asian or Pacific Islander — > GO TO 31B below 

Hispanic, regardless of race — > GO TO 31C below 

Black, not of Hispanic origin | 

White, not of Hispanic origin |-~> SKIP TO Question 32, Page 16 

American Indian or Alaskan Native | 

31 B. Which of these best categorizes your background? (MARK ONE) 

ASIAN OR PACIFIC ISLANDER 

Chinese 
Filipino 
Japanese 
Korean 

Southeast Asian (Vietnamese, Laotian, Cambodian/Kampuchean, Thai, etc.) 
Pacific Islander (Samoan, Guamanian, etc.) 

South Asian (Asian Indian, Pakistani, Bangladeshi, Sri Lankan, etc.) 
West Asian (Iranian, Afghan, Turkish, etc.) 
Middle Eastern (Iraqi, Israeli, Labanese, etc.) 
Other Asian 

31 C. Which of these best categorizes your background? (MARK ONE) 
HISPANIC 

Mexican, Mexican-American, Chicano 
Cuban 

Puerto Rican 
Other Hispanic 

31 D. What is your race? (MARK ONE) 

Black Hispanic 
White Hispanic 
Other Hispanic 
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Figure 2. Examples of Variation in Response Options for Demographic 
Items: Citizenship 



from Survey of Earned Doctorates, 1989-90 

7. Citizenship: 

□ United States, native 

□ United States, naturalized 
Non-United States 

□ Permanent Resident of United States (immigrant visa) 

□ Temporary Resident of United States (Non-immigrant visa) 



from 1990 Survey of Natural and Social Science and Engineering Graduates 
28. Are you: 



1. 


□ 


U.S. citizen 


2. 


□ 


U.S. naturalized 


3. 


□ 


Non-U.S. immigrant 






(Permanent Resident) 


4. 


□ 


Non-U.S. citizen, nonimmigrant 






(Temporary Resident) 



from 1989 Information Sheet for Former Participants in the U.S. Department 
of Energy Laboratory Graduate Thesis Research Program 

5. Citizenship 

1 . □ U.S. Native Born 

2. O U.S. Naturalized 

3. O Permanent Resident Alien 

4. □ Non-U.S. 



ERIC 



16 18 



Figure 3. Example of Item Adopted for Use in Focused Follow-Up Assessment: 
Type of Employer 



7. Which category best describes the type of your principal employment OR postdoctoral appointment 
during FEBRUARY 1989? 

0 Self-employed 

1 Business or industry 

2 Junior college, 2-year college, technical institute 

3 Medical school (including university-affiliated hospital or medical center) 

4 4-year college 

5 University, other than medical school 

6 Elementary, middle, or secondary school system 

7 Private foundation 

8 Hospital or clinic 

g u.S. military service, active duty, or Commissioned Corps, e.g., USPHS, NOAA 

10 U.S. government, civilian employee 

1 1 State government 

1 2 Local or other government, specify 

13 Nonprofit organization, other than those listed above 

1 4 Other, specify . — 



Source: 1989 Survey of Doctoral Scientists and Engineers 
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Example of Item Adopted for Use in Focused Follow-Up Assessment: 
Primary Work Activity 



1 2. From the activities listed below, select your primary and secondary work activities for your principal 
job (as reported in #6), in terms of time devoted during a typical week. 

Enter the appropriate codes (1-16) for each in the specified space. 

Primary activity Secondary activity 

1 . Teaching 

2. Basic research (i.e., study directed toward gaining scientific knowledge primarily for its 

own sake) 

3. Applied research (i.e., study directed toward gaining scientific knowledge in an effort to 

meet a recognized need) 

4. Development of equipment, products, systems 

5. Design of equipment, processes, models 

6. Management/administration of R&D 

7. Management/administration of educational/other programs 

8. Report and technical writing, editing 

9. Professional service to individuals, clinical diagnosis, psychotherapy 

10. Consulting 

1 1 . Operations-production, maintenance, construction, installation 

12. Quality control, testing, evaluation 

13. Sales, marketing, purchasing, customer and public relations 

14. Statistical work-survey work, forecasting, statistical analysis 

15. Computer applications 

1 6. Other, specify: 



Source: 1989 Survey of Doctoral Scientists and Engineers 
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Figure 5. Example of Item Adopted for Use in Focused Follow-Up Assessment: Area 
of National Interest 



18a. From this list of selected areas of national interest, indicate the ONE area to which you devoted the 
MOST professional time during a typical week at the job reported in #6. 



1 Energy and fuel 

2 Health 

3 Environment 

4 Education 

5 National defense 

6 Food or agriculture 

7 Biotechnology 

8 Mineral resources 

9 Community development and service 

10 Housing (planning, design, construction) 

11 Transportation 

12 Communications 

13 , Space 

14 None of the above 



Source: 1 989 Survey of Doctoral Scientists and Engineers 



21 

19 



Figure 6. Example of Benchmark Comparison Groups: Primary Work Activity 



Primary Work Activity of 1979-1987 Lab Grad Survey Respondents and Other Scientists 
end Engineers 





Lab Grad 
Respon- 
dents 


PhD S/Es 
of Age 
30-39 y 


All PhD 
S/£s 1 


D0E- 
Funded 
PhD S/Es 2 


AIIS/E<; 3 


Energy- 
Related 
S/Es* 


Activity . 

Applied Research 
Basic Research 
Development/Design of 

Drnrli irtc /PrflPP <5<5PS 

SUBTOTAL - R&D 


19.6% 
26.7% 
3.5% 

49.8% 


23.9% 
22.4% 
5.2% 

51.5% 


17.2% 
15.1% 
4.5% 

36.8% 


21.8% 
4.0% 

57.0% 


5 6% 
2.9% 
18.9% 

27.4% 


7.3% 
2.6% 
24.8% 

34.7% 


Computer Applications 
Consulting 

Management of Programs or R&D 
Professional Services to 
Individuals , 
Teaching 

All Others and No Response 


3.9% 
8.6% 
5.1% 
3.9% 

7.1% 
21.6% 


3.1% 
2.3% 
8.8% 
10.3% 

18.8% 
5.2% 


2.8% 
3.3% 
16.2% 
7.8% 

26.2% 
6.9% 


1.4% 
5.0% 
17.9% 
0.2% 

12.5% 
6.0% 


10.2% 
5.5% 

27.7% 
1.9% 

7.7% 
19.6% 


8.8% 
7.4% 
19.1% 

1 .3% 
28.7% 6 


TOTAL 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 



NOTES: 



1 Characteristics of Doctoral Scientists and Engineers in the United States: 1987. NSF 88-331 
2 ORAU data based on NSFs 1987 Survey of Doctorate Recipients 
3 (/.S. Scientists and Engineers: 1986, NSF 87-322 

Energy-Related Science and Engineering Personnel Outlook, 1987; DOE/OR/00033-H1 
includes Professional Services to Individuals 



Source: U.S. Department of Energy Laboratory Graduate ^search Participati on 

Program: Profile and Survey of 1979- 1087 Participants, Apnl 1 990, pp. 1 9- 
20. 
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Figure 7. Example of Benchmark Comparison Groups: Status of Federal Support 



Status of Federal Support for 1979-1987 Lab Grad Survey 
Respondents and Other Scientists and Engineers 



Status of Federal Support 


Lab Grad 
Respon- 
dents 


All PhD 
S/Es 1 


AllS/Es 2 


Energy- 
Related 
PhD S/Es 3 


Funded 
Not Funded 

Not Sure and No Response 


61.9% 
31.4% 
6.7% 


43.7% 
52.7% 
3.6% 


30.9% 
64.2% 
4.9% 


42.2% 
56.7% 
1.1% 


TOTAL 


100.0% 


100.0% 


100.0% 


100.0% 



NOTES: 

1 Characteristics of Doctoral Scientists and Engineers in the United 
States: 1987, NSF 88-331 

2 U.S. Scientists and Engineers: 1986, NSF 87-322 

^Doctoral Scientists and Engineers Working in Energy-Related 
Activities, 1985; DOE/ER-0322, May 1987 



Source: U.S. Department of Energy Laboratory Graduate Research Participation 
Program: Profile and Survey of 1979-1987 Participants, April 1990. p. 
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Figure 8. Example of Benchmark Comparison Groups: Federal Funding Source 



Federal Funding Source of 1979-1987 Lab Grad Survey Respondents and Other 
Scientists and Engineers 



Source of Federal Funds 


Lab Grad 
Respon- 
dents 


All PhD 
S/fs 1 


All S/fs 2 


Energy- 
Related 
PhD S/Es* 




Department of Defense 
Department of Energy 
Environmental Protection Agency 


17.1% 
51.3%. 
8.2% 


23.6% 
13.5% 
4.3% 


40.0% 
7.6% 
3.8% 


17.4% 
55.1% 


5 


National Institutes of Health 


1 1 .4% 


24.4% 


5.9% 3 




5 


National Science Foundation 
All Others 


10.1% 
31.6% 


17.2% 
47.5% 


3.1% 
31.6% 


11.1% 
49.6% 


6 


Uncertain of Support/Don't Know Source Agency 


1.3% 


6.0% 


13.8% 






TOTAL 


131.0% 


136.5% 


105.8% 


133.2% 





NOTES: 

Percentages are the proportion of funded S/Es who receive support from each agency. 
Totals add to more than 100% because some individuals are funded by more than one agency. 
^Characteristics of Doctoral Scientists and Engineers in the United States: 1987, NSF 88-331 
2 U.S. Scientists and Engineers: 1986, NSF 87-322 
includes all of DHHS 

^Doctoral Scientists and Engineers Working in Energy-Related Activities, 1985; 

DOE/ER-0322, May 1987 
included in All Others category 
includes Don't Know Source Agency 



Source: U.S. Department of Energy Laboratory Graduate Research Participation 

Program: Profile and Survey of 1979-1987 Participants, April 1990, p. 22. 
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Figure 9. Example of More Definitive Comparison Groups: Academic Status 



Academic Status of 
SERS Respondents 

(n=317) 




3.1% 30.0% 



Source: SERS foltow-up survey of participants during academic year 1987-1988 through 
Fall 1989. 

Academic Status of 1986-1987 
Science and Engineering 
Bachelor's Graduates 




Not a Student 
70.7% 



Source: 1988 NSF New Entrants Survey. 



Source: U.S. Department of Energy Science and Engineering Research Semester: 
Profile and Survey of 1987-1990 SERS Participants, July 1991, p. 18. 
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Figure 10. Example of More Definitive Comparison Groups: Undergraduate Majors 



Undergraduate Science and Engineering Majors: 
SERS Participants Versus Bachelor's Graduates 



Selected Fields 



Biological 
Sciences 



Chemistry 

Computer/ 
Math. ScL 

Earth/Env. 
& Marine 
Sciences 

Engineering 

Physics/ 
Astronomy 




20% 30% 40% 
Percent Distribution 



60% 



Note: These data have been restricted to include onty engineering and 
physical, biological, and computer/mathematical sciences. The 
figure represents distribution across these selected fields. 

SERS participants include those for academic years 1987-1988 
through 1989-1990. 

Bachelor's graduates data were from the 1988 New Entrants Survey 
for those who received degrees in the academic year 1986-1987. 



Source: U.S. Department of Energy Science and Engineering Research Semester: 
Profile and Survey of 1987-1990 SERS Participants, July 1991 , p. 1 3. 
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